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PATENT 

ATTORNEY DOCKET NO. 0661 12.0132 

METHOD AND DEVICE FOR MONITORING AND ANALYZING SIGNALS 

CROSS-REFERENCE TO RELATED APPLICATIONS 

This application claims the benefit of pending U.S. Patent Application Serial No. 
08/999,766, filed July 23, 1997, entitled "Steganographic Method and Device"; pending U.S. 
Patent Application Serial No. 08/772,222, filed December 20, 1996, entitled "Z-Transform 
Implementation of Digital Watermarks"; pending U.S. Patent Application Serial No. 09/456,319, 
filed December 8, 1999, entitled "Transform Implementation of Digital Watermarks"; pending 
U.S. Patent Application Serial No. 08/674,726, filed July 2, 1996, entitled "Exchange 
Mechanisms for Digital Information Packages with Bandwidth Securitization, Multichannel 
Digital Watermarks, and Key Management"; pending U.S. Patent Application Serial No. 
09/545,589, filed April 7, 2000, entitled "Method and System for Digital Watermarking"; 
pending U.S. Patent Application Serial No. 09/046,627, filed March 24, 1998, entitled "Method 
for Combining Transfer Function with Predetermined Key Creation"; pending U.S. Patent 
Application Serial No. 09/053,628, filed April 2, 1998, entitled "Multiple Transform Utilization 
and Application for Secure Digital Watermarking"; pending U.S. Patent Application Serial No. 
09/281,279, filed March 30, 1999, entitled "Optimization Methods for the Insertion, Protection, 
and Detection..."; U.S. Patent Application Serial No.09,594,719, filed June 16, 2000, entitled 
"Utilizing Data Reduction in Steganographic and Cryptographic Systems" (which is a 
continuation-in-part of PCT application No. PCT/US00/06522, filed March 14, 2000, which PCT 
application claimed priority to U.S. Provisional Application No. 60/125,990, filed March 24, 
1999); pending U.S. Application No 60/169,274, filed December 7, 1999, entitled "Systems, 
Methods And Devices For Trusted Transactions"; and PCT Application No. PCT/US00/21189, 
filed August 4, 2000 (which claims priority to U.S. Patent Application Serial No. 60/147,134, 
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filed August 4, 1999, and to US Patent Application No. 60/213,489, filed June 23, 2000, both of 
which are entitled, "A Secure Personal Content Server"). The previously identified patents and/or 
patent applications are hereby incorporated by reference, in their entireties. 

In addition, this application hereby incorporates by reference, as if fully stated herein, the 
5 total disclosures of US Patent 5,613,004 "Steganographic Method and Device"; U.S. Patent 
5,745,569 "Method for Stega-Cipher Protection of Computer Code"; and U.S. Patent 5,889,868 
"Optimization Methods for the Insertion, Protection, and Detection of Digital Watermarks in 
Digitized Data." 

li) BACKGROUND OF THE INVENTION 
^ 1 . Field of the Invention 

l f I The invention relates to the monitoring and analysis of digital information. A method 

and device are described which relate to signal recognition to enhance identification and 
^ monitoring activities. 

Q 

□ 2 . Description of the Related Art 

□ Many methods and protocols are known for transmitting data in digital form for 
multimedia applications (including computer applications delivered over public networks such as 
the internet or World Wide Web ("WWW"). These methods may include protocols for the 

20 compression of data, such that it may more readily and quickly be delivered over limited 

bandwidth data lines. Among standard protocols for data compression of digital files may be 
mentioned the MPEG compression standards for audio and video digital compression, 
promulgated by the Moving Picture Experts Group. Numerous standard reference works and 
patents discuss such compression and transmission standards for digitized information. 

25 Digital watermarks help to authenticate the content of digitized multimedia information, 

and can also discourage piracy. Because piracy is clearly a disincentive to the digital distribution 
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of copyrighted content, establishment of responsibility for copies and derivative copies of such 
works is invaluable. In considering the various forms of multimedia content, whether "master," 
stereo, NTSC video, audio tape or compact disc, tolerance of quality will vary with individuals 
and affect the underlying commercial and aesthetic value of the content. It is desirable to tie 
5 copyrights, ownership rights, purchaser information or some combination of these and related 
data into the content in such a manner that the content must undergo damage, and therefore 
reduction of its value, with subsequent, unauthorized distribution, commercial or otherwise. 
Digital watermarks address many of these concerns. A general discussion of digital 
watermarking as it has been applied in the art may be found in U.S. Patent No. 5,687,236 (whose 
ID specification is incorporated in whole herein by reference). 

:^ Further applications of basic digital watermarking functionality have also been 

j=?| developed. Examples of such applications are shown in U.S. Patent No. 5,889,868 (whose 

\Z specification is incorporated in whole herein by reference). Such applications have been drawn, 

H for instance, to implementations of digital watermarks that were deemed most suited to particular 

® transmissions, or particular distribution and storage mediums, given the nature of digitally 

Q sampled audio, video, and other multimedia works. There have also been developed techniques 

□ for adapting watermark application parameters to the individual characteristics of a given digital 

sample stream, and for implementation of digital watermarks that are feature-based - i.e., a 
system in which watermark information is not carried in individual samples, but is carried in the 
20 relationships between multiple samples, such as in a waveform shape. For instance, natural 

extensions may be added to digital watermarks that may also separate frequencies (color or 
audio), channels in 3D while utilizing discreteness in feature-based encoding only known to 
those with pseudo-random keys (i.e., cryptographic keys) or possibly tools to access such 
information, which may one day exist on a quantum level 
25 A matter of general weakness in digital watermark technology relates directly to 

the manner of implementation of the watermark. Many approaches to digital watermarking leave 
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detection and decode control with the implementing party of the digital watermark, not the 
creator of the work to be protected. This weakness removes proper economic incentives for 
improvement of the technology. One specific form of exploitation mostly regards efforts to 
obscure subsequent watermark detection. Others regard successful over encoding using the same 
5 watermarking process at a subsequent time. Yet another way to perform secure digital 
watermark implementation is through "key-based" approaches. 

SUMMARY OF THE INVENTION 

A method for monitoring and analyzing at least one signal is disclosed, which method 
JJ) comprises the steps of: receiving at least one reference signal to be monitored; creating an 
^ abstract of the at least one reference signal; storing the abstract of the at least one reference signal 

:f ; in a reference database; receiving at least one query signal to be analyzed; creating an abstract of 

the at least one query signal; and comparing the abstract of the at least one query signal to the 
^ abstract of the at least one reference signal to determine if the abstract of the at least one query 

; S signal matches the abstract of the at least one reference signal. 

□ A method for monitoring a plurality of reference signals is also disclosed, which method 

□ comprises the steps of: creating an abstract for each one of a plurality of reference signals; 
storing each of the abstracts in a reference database; receiving at least one query signal to be 
analyzed; creating an abstract of each at least one query signal; locating an abstract in the 

20 reference database that matches the abstract of each at least one query signal; and recording the 
identify of the reference signal whose abstract matched the abstract of each at least one query 
signal. 

A computerized system for monitoring and analyzing at least one signal is also disclosed, 
which system comprises: a processor for creating an abstract of a signal using selectable criteria; 
25 a first input for receiving at least one reference signal to be monitored, the first input being 

coupled to the processor such that the processor may generate an abstract for each reference 
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signal input to the processor; a reference database, coupled to the processor, for storing abstracts 
of each at least one reference signal; a second input for receiving at least one query signal to be 
analyzed, the second input being coupled to the processor such that the processor may generate 
an abstract for each query signal; and a comparing device, coupled to the reference database and 
to the second input, for comparing an abstract of the at least one query signal to the abstracts 
stored in the reference database to determine if the abstract of the at least one query signal 
matches any of the stored abstracts. 

Further, an electronic system for monitoring and analyzing at least one signal is disclosed, 
which system comprises: a first input for receiving at least one reference signal to be monitored, 
a first processor for creating an abstract of each reference signal input to the first processor 
through the first input; a second input for receiving at least one query signal to be analyzed, a 
second processor for creating an abstract of each query signal; a reference database for storing 
abstracts of each at least one reference signal; and a comparing device for comparing an abstract 
of the at least one query signal to the abstracts stored in the reference database to determine if the 
abstract of the at least one query signal matches any of the stored abstracts. 

DETAILED DESCRIPTION OF THE INVENTION 

While there are many approaches to data reduction that can be utilized, a primary concern 
is the ability to reduce the digital signal in such a manner as to retain a "perceptual relationship" 
between the original signal and its data reduced version. This relationship may either be 
mathematically discernible or a result of market-dictated needs. The purpose is to afford a more 
consistent means for classifying signals than proprietary, related text-based approaches. A 
simple analogy is the way in which a forensic investigator uses a sketch artist to assist in 
determining the identity of a human. 

In one embodiment of the invention, the abstract of a signal may be generated by the 
following steps: 1) analyze the characteristics of each signal in a group of audible/perceptible 
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variations for the same signal (e.g., analyze each of five versions of the same song — which 
versions may have the same lyrics and music but which are sung by different artists); and 2) 
select those characteristics which achieve remain relatively constant (or in other words, which 
have minimum variation) for each of the signals in the group. Optionally, the null case may be 
5 defined using those characteristics which are common to each member of the group of versions. 

Lossless and lossy compression schemes are appropriate candidates for data reduction 
technologies, as are those subset of approaches that are based on perceptual models, such as 
AAC, MP3, TwinVQ, JPEG, GIF, MPEG, etc. Where spectral transforms fail to assist in greater 
data reduction of the signal, other signal characteristics can be identified as candidates for further 
Ip data reduction. Linear predictive coding (LPC), z-transform analysis, root mean square (rms), 
;if signal to peak, may be appropriate tools to measure signal characteristics, but other approaches or 

[ fl combinations of signal characteristic analysis are contemplated. While such signal 

\Z characteristics may assist in determining particular applications of the present invention, a 

generalized approach to signal recognition is necessary to optimize the deployment and use of the 
83 present invention. 

□ Increasingly, valuable information is being created and stored in digital form. For 

o example, music, photographs and motion pictures can all be stored and transmitted as a series of 

binary digits - l's and O's. Digital techniques permit the original information to be duplicated 
repeatedly with perfect or near perfect accuracy, and each copy is perceived by viewers or 

20 listeners as indistinguishable from the original signal. Unfortunately, digital techniques also 
permit the information to be easily copied without the owner's permission. While digital 
representations of analog waveforms may be analyzed by perceptually-based or perceptually- 
limited analysis it is usually costly and time-consuming to model the processes of the highly 
effective ability of humans to identify and recognize a signal. In those applications where analog 

25 signals require analysis, the cost of digitizing the analog signal is minimal when compared to the 
benefits of increased accuracy and speed of signal analysis and monitoring when the processes 
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contemplated by this invention are utilized. 

The present invention relates to identification of digitally-sampled information, such as 
images, audio and video. Traditional methods of identification and monitoring of those signals 
do not rely on "perceptual quality," but rather upon a separate and additional signal. Within this 
application, such signals will be called "additive signals" as they provide information about the 
original images, audio or video, but such information is in addition to the original signal. One 
traditional, text-based additive signal is title and author information. The title and author, for 
example, is information about a book, but it is in addition to the text of the book. If a book is 
being duplicated digitally, the title and author could provide one means of monitoring the number 
of times the text is being duplicated, for example, through an Internet download. The present 
invention, however, is directed to the identification of a digital signal — whether text, audio, or 
video — using only the digital signal itself and then monitoring the number of times the signal is 
duplicated. Reliance on an additive signal has many shortcomings. For example, first, someone 
must incorporate the additive signal within the digital data being transmitted, for example, by 
concatenation or through an embedding process. Such an additive signal, however, can be easily 
identified and removed by one who wants to utilize the original signal without paying for its 
usage. If the original signal itself is used to identify the content, an unauthorized user could not 
avoid payment of a royalty simply by removing the additive signal — because there is no additive 
signal to remove. Hence, the present invention avoids a major disadvantage of the prior art. 

One such additive signal that may be utilized is a digital watermark — which ideally 
cannot be removed without perceptually altering the original signal. A watermark may also be 
used as a monitoring signal (for example, by encoding an identifier that uniquely identifies the 
original digital signal into which the identifier is being embedded). A digital watermark used for 
monitoring is also an additive signal, and such a signal may make it difficult for the user who 
wants to duplicate a signal without paying a royalty — mainly by degrading the perceptual quality 
of the original signal if the watermark (and hence the additive monitoring signal) is removed. 
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This is, however, is a different solution to the problem. 

The present invention eliminates the need of any additive monitoring signal because the 
present invention utilizes the underlying content signal as the identifier itself. Nevertheless, the 
watermark may increase the value of monitoring techniques by increasing the integrity of the 
5 embedded data and by indicating tampering of either the original content signal or the monitoring 
signal. Moreover, the design of a watermarking embedding algorithm is closely related to the 
perceptibility of noise in any given signal and can represent an ideal subset of the original signal: 
the watermark bits are an inverse of the signal to the extent that lossy compression schemes, 
which can be used, for instance, to optimize a watermarking embedding scheme, can yield 
K) information about the extent to which a data signal can be compressed while holding steadfast to 
^ the design requirement that the compressed signal maintain its perceptual relationship with the 

| J J original, uncompressed signal By describing those bits that are candidates for imperceptible 

embedding of watermark bits, further data reduction may be applied on the candidate watermarks 
as an example of retaining a logical and perceptible relationship with the original uncompressed 
© signal. 

□ Of course, the present invention may be used in conjunction with watermarking 

□ technology (including the use of keys to accomplish secure digital watermarking), but 
watermarking is not necessary to practice the present invention. Keys for watermarking may 
have many forms, including: descriptions of the original carrier file formatting, mapping of 

20 embedded data (actually imperceptible changes made to the carrier signal and referenced to the 
predetermined key or key pairs), assisting in establishing the watermark message data integrity 
(by incorporation of special one way functions in the watermark message data or key), etc. 
Discussions of these systems in the patents and pending patent applications are incorporated by 
reference above. The "recognition" of a particular signal or an instance of its transmission, and 

25 its monitoring are operations that may be optimized through the use of digital watermark 
analysis. 
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A practical difference between the two approaches of using a separate, additive 
monitoring signal and using the original signal itself as the monitoring signal is control. If a 
separate signal is used for monitoring, then the originator of the text, audio or video signal being 
transmitted and the entity doing the monitoring have to agree as to the nature of the separate 
signal to be used for monitoring — otherwise, the entity doing the monitoring would not know 
where to look, for what to look, or how to interpret the monitoring signal once it was identified 
and detected. On the other hand, if the original signal is used itself as a monitoring signal, then 
no such agreement is necessary. Moreover, a more logical and self-sufficient relationship 
between the original and its data-reduced abstract enhances the transparency of any resulting 
monitoring efforts. The entity doing the monitoring is not looking for a separate, additive 
monitoring system, and further, need not have to interpret the content of the monitoring signal. 

Monitoring implementations can be handled by robust watermark techniques (those 
techniques that are able to survive many signal manipulations but are not inherently "secure" for 
verification of a carrier signal absent a logically-related watermarking key) and forensic 
watermark techniques (which enable embedding of watermarks that are not able to survive 
perceptible alteration of the carrier signal and thus enable detection of tampering with the 
originally watermarked carrier signal). The techniques have obvious trade-offs between speed, 
performance and security of the embedded watermark data. 

In other disclosures, we suggest improvements and implementations that relate to digital 
watermarks in particular and embedded signaling in general. A digital watermark may be used to 
"tag" content in a manner that is not humanly-perceptible, in order to ensure that the human 
perception of the signal quality is maintained. Watermarking, however, must inherently alter at 
least one data bit of the original signal to represent a minimal change from the original signal's 
"unwatermarked state." The changes may affect only a bit, at the very least, or be dependent on 
information hiding relating to signal characteristics, such as phase information, differences 
between digitized samples, root mean square (RMS) calculations, z-transform analysis, or similar 
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signal characteristic category. 

There are weaknesses in using digital watermark technology for monitoring purposes. 
One weakness relates directly to the way in which watermarks are implemented. Often, the 
persons responsible for encoding and decoding the digital watermark are not the creator of the 
valuable work to be protected. As such, the creator has no input on the placement of the 
monitoring signal within the valuable work being protected. Hence, if a user wishing to avoid 
payment of the royalty can find a way to decode or remove the watermark, or at least the 
monitoring signal embedded in the watermark, then the unauthorized user may successfully 
duplicate the signal with impunity. This could occur, for example, if either of the persons 
responsible for encoding or decoding were to have their security compromised such that the 
encoding or decoding algorithms were discovered by the unauthorized user. 

With the present invention, no such disadvantages exist because the creator need not rely 
on anyone to insert a monitoring signal — as no such signal is necessary. Instead, the creator's 
work itself is used as the monitoring signal. Accordingly, the value in the signal will have a 
strong relationship with its recognizability. 

By way of improving methods for efficient monitoring as well as effective confirmation 
of the identity of a digitally-sampled signal, the present invention describes useful methods for 
using digital signal processing for benchmarking a novel basis for differencing signals with 
binary data comparisons. These techniques may be complemented with perceptual techniques, 
but are intended to leverage the generally decreasing cost of bandwidth and signal processing 
power in an age of increasing availability and exchange of digitized binary data. 

So long as there exist computationally inexpensive ways of identifying an entire signal 
with some fractional representation or relationship with the original signal, or its perceptually 
observable representation, we envision methods for faster and more accurate auditing of signals 
as they are played, distributed or otherwise shared amongst providers (transmitters) and 
consumers (receivers). The ability to massively compress a signal to its essence — which is not 
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strictly equivalent to "lossy" or "lossless" compression schemes or perceptual coding techniques, 
but designed to preserve some underlying "aesthetic quality" of the signal — represents a useful 
means for signal analysis in a wide variety of applications. The signal analysis, however, must 
maintain the ability to distinguish the perceptual quality of the signals being compared. For 
example, a method which analyzed a portion of a song by compressing it to a single line of lyrics 
fails to maintain the ability to distinguish the perceptual quality of the songs being compared. 
Specifically, for example, if the song "New York State of Mind" were compressed to the lyrics 
"Pm in a New York State of Mind," such a compression fails to maintain the ability to 
distinguish between the various recorded versions of the song, say, for example between Billy 
Joel's recording and Barbara Streisand's recording. Such a method is, therefore, incapable of 
providing accurate monitoring of the artist's recordings because it could not determine which of 
the two artists is deserving of a royalty — unless of course, there is a separate monitoring signal to 
provide the name of the artist or other information sufficient to distinguish the two versions. The 
present invention, however, aims to maintain some level of perceptual quality of the signals 
being compared and would deem such a compression to be excessive. 

This analogy can be made clearer if it is understood that there are a large number of 
approaches to compressing a signal to, say, 1/1 0,000 th of its original size, not for maintaining its 
signal quality to ensure computational ease for commercial quality distribution, but to assist in 
identification, analysis or monitoring of the signal. Most compression is either lossy or lossless 
and is designed with psychoacoustic or psychovisual parameters. That is to say, the signal is 
compressed to retain what is "humanly-perceptible." As long as the compression successfully 
mimics human perception, data space may be saved when the compressed file is compared to the 
uncompressed or original file. While psychoacoustic and psychovisual compression has some 
relevance to the present invention, additional data reduction or massive compression is 
anticipated by the present invention. It is anticipated that the original signal may be compressed 
to create a realistic or self-similar representation of the original signal, so that the compressed 
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signal can be referenced at a subsequent time as unique binary data that has computational 
relevance to the original signal Depending on the application, general data reduction of the 
original signal can be as simple as massive compression or may relate to the watermark encoding 
envelope parameter (those bits which a watermarking encoding algorithm deem as candidate bits 
for mapping independent data or those bits deemed imperceptible to human senses but detectable 
to a watermark detection algorithm). In this manner, certain media which are commonly known 
by signal characteristics, a painting, a song, a TV commercial, a dialect, etc., may be analyzed 
more accurately, and perhaps, more efficiently than a text-based descriptor of the signal. So long 
as the sender and receiver agree that the data representation is accurate, even insofar as the data- 
reduction technique has logical relationships with the perceptibility of the original signal, as they 
must with commonly agreed to text descriptors, no independent cataloging is necessary. 

The present invention generally contemplates a signal recognition system that has at least 
five elements. The actual number of elements may vary depending on the number of domains in 
which a signal resides (for example, audio is at least one domain while visual carriers are at least 
two dimensional). The present invention contemplates that the number of elements will be 
sufficient to effectively and efficiently meet the demands of various classes of signal recognition. 
The design of the signal recognition that may be used with data reduction is better understood in 
the context of the general requirements of a pattern or signal recognition system. 

The first element is the reference database, which contains information about a plurality 
of potential signals that will be monitored. In one form, the reference database would contain 
digital copies of original works of art as they are recorded by the various artists, for example, 
contain digital copies of all songs that will be played by a particular radio station. In another 
form, the reference database would contain not perfect digital copies of original works of art, but 
digital copies of abstracted works of art, for example, contain digital copies of all songs that have 
been preprocessed such that the copies represent the perceptual characteristics of the original 
songs. In another form, the reference database would contain digital copies of processed data 
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files, which files represent works of art that have been preprocessed in such a fashion as to 
identify those perceptual differences that can differentiate one version of a work of art from 
another version of the same work of art, such as two or more versions of the same song, but by 
different artists. These examples have obvious application to visually communicated works such 
as images, trademarks or photographs, and video as well 

The second element is the object locator, which is able to segment a portion of a signal 
being monitored for analysis (i.e., the "monitored signal"). The segmented portion is also 
referred to as an "object." As such, the signal being monitored may be thought of comprising a 
set of objects. A song recording, for example, can be thought of as having a multitude of objects. 
The objects need not be of uniform length, size, or content, but merely be a sample of the signal 
being monitored. Visually communicated informational signals have related objects; color and 
size are examples. 

The third element is the feature selector, which is able to analyze a selected object and 
identify perceptual features of the object that can be used to uniquely describe the selected object. 
Ideally, the feature selector can identify all, or nearly all, of the perceptual qualities of the object 
that differentiate it from a similarly selected object of other signals. Simply, a feature selector 
has a direct relationship with the perceptibility of features commonly observed. Counterfeiting is 
an activity which specifically seeks out features to misrepresent the authenticity of any given 
object. Highly granular, and arguably successful, counterfeiting is typically sought for objects 
that are easily recognizable and valuable, for example, currency, stamps, and trademarked or 
copyrighted works and objects that have value to a body politic. 

The fourth element is the comparing device which is able to compare the selected object 
using the features selected by the feature selector to the plurality of signals in the reference 
database to identify which of the signals matches the monitored signal. Depending upon how the 
information of the plurality of signals is stored in the reference database and depending upon the 
available computational capacity (e.g., speed and efficiency), the exact nature of the comparison 
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will vary. For example, the comparing device may compare the selected object directly to the 
signal information stored in the database. Alternatively, the comparing device may need to 
process the signal information stored in the database using input from the feature selector and 
then compare the selected object to the processed signal information. Alternatively, the 
comparing device may need to process the selected object using input from the feature selector 
and then compare the processed selected object to the signal information. Alternatively, the 
comparing device may need to process the signal information stored in the database using input 
from the feature selector, process the selected object using input from the feature selector, and 
then compare the processed selected object to the processed signal information. 

The fifth element is the recorder which records information about the number of times a 
given signal is analyzed and detected. The recorder may comprise a database which keeps track 
of the number of times a song, image, or a movie has been played, or may generate a serial output 
which can be subsequently processed to determine the total number of times various signals have 
been detected. 

Other elements may be added to the system or incorporated into the five elements 
identified above. For example, an error handler may be incorporated into the comparing device. 
If the comparing device identifies multiple signals which appear to contain the object being 
sought for analysis or monitoring, the error handler may offer further processing in order to 
identify additional qualities or features in the selected object such that only one of the set of 
captured signals is found to contain the further analyzed selected object that actually conforms 
with the object thought to have been transmitted or distributed. 

Moreover, one or more of the five identified elements may be implemented with software 
that runs on the same processor, or which uses multiple processors. In addition, the elements 
may incorporate dynamic approaches that utilize stochastic, heuristic, or experience-based 
adjustments to refine the signal analysis being conducted within the system, including, for 
example, the signal analyses being performed within the feature selector and the comparing 
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device. This additional analyses may be viewed as filters that are designed to meet the 
expectations of accuracy or speed for any intended application. 

Since maintenance of original signal quality is not required by the present invention, 
increased efficiencies in processing and identification of signals can be achieved. The present 
invention concerns itself with perceptible relationships only to the extent that efficiencies can be 
achieved both in accuracy and speed with enabling logical relationships between an original 
signal and its abstract. 

The challenge is to maximize the ability to sufficiently compress a signal to both retain its 
relationship with the original signal while reducing the data overhead to enable more efficient 
analysis, archiving and monitoring of these signals. In some cases, data reduction alone will not 
suffice: the sender and receiver must agree to the accuracy of the recognition. In other cases, 
agreement will actually depend on a third party who authored or created the signal in question. A 
digitized signal may have parameters to assist in establishing more accurate identification, for 
example, a "signal abstract" which naturally, or by agreement with the creator, the copyright 
owner or other interested parties, can be used to describe the original signal. By utilizing less 
than the original signal, a computationally inexpensive means of identification can be used. As 
long as a realistic set of conditions can be arrived at governing the relationship between a signal 
and its data reduced abstract, increases in effective monitoring and transparency of information 
data flow across communications channels is likely to result. This feature is significant in that it 
represents an improvement over how a digitally-sampled signal can be cataloged and identified, 
though the use of a means that is specifically selected based upon the strengths of a general 
computing device and the economic needs of a particular market for the digitized information 
data being monitored. The additional benefit is a more open means to uniformly catalog, 
analyze, and monitor signals. As well, such benefits can exist for third parties, who have a 
significant interest in the signal but are not the sender or receiver of said information. 

As a general improvement over the art, the present invention incorporates what could best 
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be described as "computer-acoustic" and "computer-visual" modeling, where the signal abstracts 
are created using data reduction techniques to determine the smallest amount of data, at least a 
single bit, which can represent and differentiate two digitized signal representations for a given 
predefined signal set. Each of such representations must have at least a one bit difference with 
all other members of the database to differentiate each such representation from the others in the 
database. The predefined signal set is the object being analyzed. The signal identifier/detector 
should receive its parameters from a database engine. The engine will identify those 
characteristics (for example, the differences) that can be used to distinguish one digital signal 
from all other digital signals that are stored in its collection. For those digital signals or objects 
which are seemingly identical, excepting that the signal may have different performance or 
utilization in the newly created object, benefits over additive or text-based identifiers are 
achieved. Additionally, decisions regarding the success or failure of an accurate detection of any 
given object may be flexibly implemented or changed to reflect market-based demands of the 
engine. Appropriate examples are songs or works or art which have been sampled or re- 
produced by others who are not the original creator. 

In some cases, the engine will also consider the NULL case for a generalized item not in 
its database, or perhaps in situations where data objects may have collisions. For some 
applications, the NULL case is not necessary, thus making the whole system faster. For instance, 
databases which have fewer repetitions of objects or those systems which are intended to 
recognize signals with time constraints or capture all data objects. Greater efficiency in 
processing a relational database can be obtained because the rules for comparison are selected for 
the maximum efficiency of the processing hardware and/or software, whether or not the 
processing is based on psychoacoustic or psychovisual models. The benefits of massive data 
reduction, flexibility in constructing appropriate signal recognition protocols and incorporation of 
cryptographic techniques to further add accuracy and confidence in the system are clearly 
improvements over the art. For example, where the data reduced abstract needs to have further 
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uniqueness, a hash or signature may be required. And for objects which have further uniqueness 
requirements, two identical instances of the object could be made unique with cryptographic 
techniques. 

Accuracy in processing and identification may be increased by using one or more of the 
following fidelity evaluation functions: 

1) RMS (root mean square). For example, a RMS function may be used to assist 
in determining the distance between data based on mathematically determinable 
Euclidean distance between the beginning and end data points (bits) of a particular 
signal carrier. 

2) Frequency weighted RMS. For example, different weights may be applied to 
different frequency components of the carrier signal before using RMS. This 
selective weighting can assist in further distinguishing the distance between beginning 
and end points of the signal carrier (at a given point in time, described as bandwidth, 
or the number of total bits that can be transmitted per second) and may be considered 
to be the mathematical equivalent of passing a carrier signal difference through a data 
filter and figuring the average power in the output carrier. 

3) Absolute error criteria, including particularly the NULL set (described above) 
The NULL may be utilized in two significant cases: First, in instances where the 
recognized signal appears to be an identified object which is inaccurately attributed or 
identified to an object not handled by the database of objects; and second, where a 
collision of data occurs. For instance, if an artist releases a second performance of a 
previously recorded song, and the two performances are so similar that their 
differences are almost imperceptible, then the previously selected criteria may not be 
able to differentiate the two recordings. Hence, the database must be "recalibrated" to 
be able to differentiate these two versions. Similarly, if the system identifies not one, 
but two or more, matches for a particular search, then the database may need 
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"recalibration" to further differentiate the two objects stored in the database. 
4) Cognitive Identification. For example, the present invention may use an 

experience-based analysis within a recognition engine. Once such analysis may 
involve mathematically determining a spectral transform or its equivalent of the 
carrier signal. A spectral transform enables signal processing and should maintain, 
for certain applications, some cognitive or perceptual relationship with the original 
analog waveform. As a novel feature to the present invention, additional classes may 
be subject to humanly-perceptible observation. For instance, an experience-based 
criteria which relates particularly to the envisioned or perceived accuracy of the data 
information object as it is used or applied in a particular market, product, or 
implementation. This may include a short 3 second segment of a commercially 
available and recognizable song which is used for commercials to enable recognition 
of the good or service being marketed. The complete song is marketed as a separately 
valued object from the use of a discrete segment of the song (that may be used for 
promotion or marketing — for the complete song or for an entirely different good or 
service). To the extent that an owner of the song in question is able to further enable 
value through the licensing or agreement for use of a segment of the original signal, 
cognitive identification is a form of filtering to enable differentiations between 
different and intended uses of the same or subset of the same signal (object). The 
implementation relating specifically, as disclosed herein, to the predetermined 
identification or recognition means and/or any specified relationship with subsequent 
use of the identification means can be used to create a history as to how often a 
particular signal is misidentified, which history can then be used to optimize 
identification of that signal in the future. The difference between use of an excerpt of 
the song to promote a separate and distinct good or service and use of the excerpt to 
promote recognition of the song itself (for example, by the artist to sell copies of the 
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song) relates informationally to a decision based on recognized and approved use of 
the song. Both the song and applications of the song in its entirety or as a subset are 
typically based on agreement by the creator and the sender who seeks to utilize the 
work. Trust in the means for identification, which can be weighted in the present 
invention (for example, by adjusting bit-addressable information), is an important 
factor in adjusting the monitoring or recognition features of the object or carrier 
signal, and by using any misidentification information, (including any experience- 
based or heuristic information), additional features of the monitored signal can be 
used to improve the performance of the monitoring system envisioned herein. The 
issue of central concern with cognitive identification is a greater understanding of the 
parameters by which any given object is to be analyzed. To the extent that a creator 
chooses varying and separate application of his object, those applications having a 
cognitive difference in a signal recognition sense (e.g., the whole or an excerpt), the 
system contemplated herein includes rules for governing the application of bit- 
addressable information to increase the accuracy of the database. 
5) Finally, the predetermined parameters that are associated with a discrete case 

for any given object will have a significant impact upon the ability to accurately 
process and identify the signals. For example, if a song is transmitted over a FM 
carrier, then one skilled in the art will appreciate that the FM signal has a 
predetermined bandwidth which is different from the bandwidth of the original 
recording, and different even from song when played on an AM carrier, and different 
yet from a song played using an 8 -bit Internet broadcast. Recognition of these 
differences, however, will permit the selection of an identification means which can 
be optimized for monitoring a FM broadcasted signal. In other words, the 
discreteness intended by the sender is limited and directed by the fidelity of the 
transmission means. Objects may be cataloged and assessing with the understanding 
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that all monitoring will occur using a specific transmission fidelity. For example, a 
database may be optimized with the understanding that only AM broadcast signals 
will be monitored. For maximum efficiency, different data bases may be created for 
different transmission channels, e.g., AM broadcasts, FM broadcasts, Internet 
broadcasts, etc. 

For more information on increasing efficiencies for information systems, see The Mathematical 
Theory of Communication (1948), by Shannon. 

Because bandwidth (which in the digital domain is equated to the total number of bits that 
can be transmitted in a fixed period of time) is a limited resource which places limitations upon 
transmission capacity and information coding schemes, the importance of monitoring for 
information objects transmitted over any given channel must take into consideration the nature 
and utilization of a given channel. The supply and demand of bandwidth will have a dramatic 
impact on the transmission, and ultimately, upon the decision to monitor and recognize signals. 
A discussion of this is found in a co-pending application by the inventor under U.S. Patent 
Application No. 08/674,726 "Exchange Mechanisms for Digital Information Packages with 
Bandwidth Securitization, Multichannel Digital Watermarks, and Key Management" (which 
application is incorporated herein by reference as if fully setforth herein). 

If a filter is to be used in connection with the recognition or monitoring engine, it may be 
desirable for the filter to anticipate and take into consideration the following factors, which affect 
the economics of the transmission as they relate to triggers for payment and/or relate to events 
requiring audits of the objects which are being transmitted: 1) time of transmission (i.e., the point 
in time when the transmission occurred), including whether the transmission is of a live 
performance); 2) location of transmission (e.g., what channel was used for transmission, which 
usually determines the associated cost for usage of the transmission channel); 3) the point of 
origination of the transmission (which may be the same for a signal carrier over many distinct 
channels); and 4) pre-existence of the information carrier signal (pre-recorded or newly created 
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information carrier signal, which may require differentiation in certain markets or instances). 

In the case of predetermined carrier signals (those which have been recorded and stored 
for subsequent use), "positional information carrier signals" are contemplated by this invention, 
namely, perceptual differences between the seemingly "same" information carrier that can be 
5 recognized as consumers of information seek different versions or quality levels of the same 
carrier signal. Perceptual differences exist between a song and its reproduction from a CD, an 
AM radio, and an Internet broadcast. To the extent that the creator or consumer of the signal can 
define a difference in any of the four criteria above, means can be derived (and programmed for 
selectability) to recognize and distinguish these differences. It is, however, quite possible that 
W the ability to monitor carrier signal transmission with these factors will increase the variety and 
;jf richness of available carrier signals to existing communications channels. The differentiation 
^] between an absolute case for transmission of an object, which is a time dependent event, for 

instance a live or real time broadcast, versus the relative case, which is prerecorded or stored for 
^ transmission at a later point in time, creates recognizable differences for signal monitoring. 

© The monitoring and analysis contemplated by this invention may have a variety of 

□ purposes, including, for example, the following: to determine the number of times a song is 

□ broadcast on a particular radio broadcast or Internet site; to control security though a voice- 
activated security system; and to identify associations between a beginner's drawing and those of 
great artists (for example to draw comparisons between technique, compositions, or color 

20 schemes). None of these examples could be achieved with any significant degree of accuracy 
using a text-based analysis. Additionally, strictly text-based systems fail to fully capture the 
inherent value of the data recognition or monitoring information itself. 

SAMPLE EMBODIMENTS 
25 In order to better appreciate and understand the present invention, the following sample 

embodiments are provided. These sample embodiments are provided for exemplary purposes 
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only, and in no way limit the present invention. 



SAMPLE EMBODIMENT 1 

A database of audio signals (e.g., songs) is stored or maintained by a radio station or 
5 Internet streaming company, who may select a subset of the songs are stored so that the subset 

may be later broadcast to listeners. The subset, for example, may comprise a sufficient number 
of songs to fill 24 hours of music programming (between 300 or 500 songs). Traditionally, 
monitoring is accomplished by embedding some identifier into the signal, or affixing the 
identifier to the signal, for later analysis and determination of royalty payments. Most of the 
L0 traditional analysis is performed by actual persons who use play lists and other statistical 
^ approximations of audio play, including for example, data obtained through the manual (i.e., by 

; sp j persons) monitoring of a statistically significant sample of stations and transmission times so that 

! "7 an extrapolation may be made to a larger number of comparable markets. 

The present invention creates a second database from the first database, wherein each of 
© the stored audio signals in the first database is data reduced in a manner that is not likely to 
O reflect the human perceptual quality of the signal, meaning that a significantly data-reduced 

□ signal is not likely to be played back and recognized as the original signal. As a result of the data 

reduction, the size of the second database (as measured in digital terms) is much smaller than the 
size of the first database, and is determined by the rate of compression. If, for example, if 24 
20 hours worth of audio signals are compressed at a 10,000:1 compression rate, the reduced data 
could occupy a little more than 1 megabyte of data. With such a large compression rate, the data 
to be compared and/or analyzed may become computationally small such that computational 
speed and efficiency are significantly improved. 

With greater compression rates, it is anticipated that similarity may exist between the data 
25 compressed abstractions of different analog signals (e.g., recordings by two different artists of the 

same song). The present invention contemplates the use of bit-addressable differences to 
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distinguish between such cases. In applications where the data to be analyzed has higher value in 
some predetermined sense, cryptographic protocols, such as a hash or digital signature, can be 
used to distinguish such close cases. 

In a preferred embodiment, the present invention may utilize a centralized database where 
5 copies of new recordings may be deposited to ensure that copyright owners, who authorize 

transmission or use of their recordings by others, can independently verify that the object is 
correctly monitored. The rules for the creator himself to enter his work would differ from a 
universally recognized number assigned by an independent authority (say, ISRC, ISBN for 
recordings and books respectively). Those skilled in the art of algorithmic information theory 
IQ (AIT) can recognize that it is now possible to describe optimized use of binary data for content 
and functionality. The differences between objects must relate to decisions made by the user of 
Z\ the data, introducing subjective or cognitive decisions to the design of the contemplated 
\Z invention as described above. To the extent that objects can have an optimized data size when 

|saa compared with other objects for any given set of objects, the algorithms for data reduction would 
S have predetermined flexibility directly related to computational efficiency and the set of objects 

□ to be monitored. The flexibility in having transparent determination of unique signal abstracts, 

□ as opposed to independent third party assignment, is likely to increase confidence in the 
monitoring effort by the owners of the original signals themselves. The prior art allows for no 
such transparency to the copyright creators. 

20 

SAMPLE EMBODIMENT 2 

Another embodiment of the invention relates to visual images, which of course, involve 
at least two dimensions. 

Similar to the goals of a psychoacoustic model, a psychovisual model attempts to 
25 represent a visual image with less data, and yet preserve those perceptual qualities that permit a 
human to recognize the original visual image. Using the very same techniques described above 
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in connection with an audio signal, signal monitoring of visual images may be implemented. 

One such application for monitoring and analyzing visual images involves a desire to find 
works of other artists that relate to a particular theme. For example, finding paintings of sunsets 
or sunrises. A traditional approach might involve a textual search involving a database wherein 
5 the works of other artists have been described in writing. The present invention, however, 
involves the scanning of an image involving a sun, compressing the data to its essential 
characteristics (i.e., those perceptual characteristics related to the sun) and then finding matches 
in a database of other visual images (stored as compressed or even uncompressed data). By 
studying the work of other artists using such techniques, a novice, for example, could learn much 
m by comparing the presentations of a common theme by different artists. 

™ Another useful application involving this type of monitoring and analyzing is the 

!;M identification of photographs of potential suspects whose identity matches the sketch of a police 

\Z artist. 

^ Note that combinations of the monitoring techniques discussed above can be used for 

B audio-visual monitoring, such as video-transmission by a television station or cable station. The 

□ techniques would have to compensate, for example, for a cable station that is broadcasting a 
Q audio channel unaccompanied by video. 

Other embodiments and uses of the invention will be apparent to those skilled in the art 
from consideration of the specification and practice of the invention disclosed herein. The 
20 specification and examples should be considered exemplary only with the true scope and spirit of 

the invention indicated by the following claims. As will be easily understood by those of 
ordinary skill in the art, variations and modifications of each of the disclosed embodiments can 
be easily made within the scope of this invention as defined by the following claims. 
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WHAT IS CLAIMED IS: 

1 . A method for monitoring and analyzing at least one signal comprising: 
receiving at least one reference signal to be monitored; 

creating an abstract of said at least one reference signal; 

storing the abstract of said at least one reference signal in a reference database; 

receiving at least one query signal to be analyzed; 

creating an abstract of said at least one query signal; 

comparing the abstract of said at least one query signal to the abstract of said at least 
one reference signal to determine if the abstract of said at least one query signal matches the 
abstract of said at least one reference signal. 

2. The method of claim 1 wherein 

the step of creating an abstract of said at least one reference signal comprises: 
inputting the reference signal to a processor; 

creating an abstract of the reference signal using perceptual qualities of the 
reference signal such that the abstract retains a perceptual relationship to the reference 
signal from which it is derived; and 

the step of creating an abstract of said at least one query signal comprises: 
inputting the at least one query signal to the processor; 

creating an abstract of the at least one query signal using perceptual qualities of 
the at least one query signal such that the abstract retains a perceptual relationship to 
the at least one query signal from which it is derived. 

3. The method of claim 1 further comprising: 

creating at least one counter corresponding to one of said at least one reference signals, 
said at least one counter being representative of the number of times a match is found between 
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the abstract of said at least one query signal and the abstract of said at least one reference 
signal; and 

incrementing the counter corresponding to a particular reference signal when a match 
is found between an abstract of said at least one query signal and the abstract of the particular 
reference signal. 

4. The method of claim 1 further comprising: 

recording an occurrence of a match between the abstract of said at least one query 
signal and the abstract of said at least one reference signal; and 

generating a report that identifies the reference signal whose abstract matched the 
abstract of said at least one query signal 

5 . The method of claim 4, further comprising: 

recording an occurrence of a match between the abstract of said at least one query 
signal and the abstract of said at least one reference signal. 

6. The method of claim 1, further comprising permitting access to a secured area when 
the abstract of said at least one query signal matches the abstract of said at least one reference 
signal. 

7. The method of claim 1 , wherein 

the step of creating an abstract of said at least one reference signal comprises: 

using a portion of said at least one reference signal to create an abstract of said 
at least one reference signal; and 

the step of creating an abstract of said at least one query signal comprises: 

using a portion of said at least one query signal to create an abstract of said at 
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least one query signal. 

8. A method for monitoring a plurality of reference signals, comprising: 
creating an abstract for each of the plurality of reference signals; 
storing each of said abstracts in a reference database; 

receiving at least one query signal to be analyzed; 

creating an abstract of each of the at least one query signals; 

locating an abstract in the reference database that matches the abstract of each at least 
one query signal; and 

recording the identify of the reference signal whose abstract matched the abstract of 
each at least one query signal. 

9. The method of claim 8, wherein 

the step of creating an abstract for each of a plurality of reference signals comprises: 

inputting each of the plurality of reference signals to a processor; 

creating an abstract of each one of the plurality of reference signals using 
perceptual qualities of each one of a plurality of reference signals such that the abstract 
retains a perceptual relationship to the reference signal from which it is derived; and 
the step of creating an abstract of each of the at least one query signals comprises: 

inputting each of the at least one query signals to a processor; 

creating an abstract of each one of a plurality of reference signals using 
perceptual qualities of each one of a plurality of reference signals such that the abstract 
retains a perceptual relationship to the reference signal from which it is derived 



1 0. The method of claim 8, wherein 

the step of creating an abstract of said at least one reference signal comprises: 
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using a portion of said at least one reference signal to create an abstract 
of said at least one reference signal; 

and the step of creating an abstract of said at least one query signal comprises: 
using a portion of said at least one query signal to create an abstract of 
5 said at least one query signal. 

1 1 . The method of claim 8 , further comprising: 

creating at least one counter corresponding to one of said plurality of reference signals, 

said at least one counter being representative of the number of times a match is found between 
M the abstract of said at least one query signal and an abstract of one of said plurality of 

S reference signals; and 

: J j incrementing the counter corresponding to a particular reference signal when a match 

;;i is found between an abstract of said at least one query signal and the abstract of the particular 

j5dfe reference signal. 

! # 

□ 12. The method of claim 8, further comprising permitting access to a secured area when 

Q the abstract of said at least one query signal matches an abstract of one of said plurality of 

reference signals. 

20 13. A computerized system for monitoring and analyzing at least one signal: 

a processor that creates an abstract of a signal using selectable criteria; 
a first input that receives at least one reference signal to be monitored, said first input 
being coupled to said processor such that said processor may generate an abstract for each 
reference signal input to said processor; 
25 a reference database, coupled to said processor, that stores abstracts of each at least 

one reference signal; 
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a second input that receives at least one query signal to be analyzed, said second input 
being coupled to said processor such that said processor may generate an abstract for each 
query signal; 

a comparing device, coupled to said reference database and to said second input, that 
5 compares an abstract of said at least one query signal to the abstracts stored in the reference 

database to determine if the abstract of said at least one query signal matches any of the stored 
abstracts, 

14, The system of claim 13, further comprising: 

M a storage medium coupled to said first input, that stores each of said at least one 

j'S reference signals to be monitored; and 

if I a controller coupled to the first input, the processor, the comparing device, the 

Jl reference database and the storage medium, said controller causing an abstract for each 

^ reference signal being input for the first time to be compared to all previously stored abstracts 

in the reference database, such that in the event that the comparing device determines that it 

□ cannot distinguish between the abstract of a reference signal being input for the first time from 

□ a previously stored abstract in the reference database, the controller adjusts the criteria being 
used by the processor and re-generates the reference database, by re-processing each reference 
signal stored on the storage medium to create new abstracts and storing said new abstracts in 

20 the reference database. 

15. The system of claim 14, wherein the controller includes a means to adjust compression 
rates at which the processor processes a signal to create an abstract. 

25 16. The system of claim 13, wherein the comparing device identifies at least two abstracts 

in the reference database that match the abstract of said at least one query signal and an index 
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of relatedness to said at least one query signal for each of said at least two matching abstracts, 

17. The system of claim 13, further comprising: 

a security controller that controls access to a secured area, such that access is granted 
only if the comparing device confirms that an abstract of said at least one query signal 
matches an abstract of said at least one reference signal 

18. The system of claim 13, wherein said first input and said second input are the same. 

19. The system of claim 13, wherein said second input is remotely coupled to the 
processor. 

20. The system of claim 13, further comprising: 

a recorder that records the identify of the reference signal whose abstract matched the 
abstract of said at least one query signal; and 

a report generator that generates a report that identifies the reference signals whose 
abstracts matched the abstract of said at least one query signal. 

21. A electronic system for monitoring and analyzing at least one signal, comprising: 
a first input that receives at least one reference signal to be monitored, 

a first processor that creates an abstract of each reference signal input to said first 
processor through said first input; 

a second input that receives at least one query signal to be analyzed, 

a second processor that creates an abstract of each query signal; 

a reference database that stores abstracts of each at least one reference signal; 

a comparing device that compares an abstract of said at least one query signal to the 
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abstracts stored in the reference database to determine if the abstract of said at least one query 
signal matches any of the stored abstracts. 

22. The system of claim 2 1 , wherein said second input is remotely coupled to the system. 

23. The system of claim 21 , wherein said second processor is remotely coupled to the 
system. 

24. The system of claim 21, wherein the system transmits the criteria that are being used 
by the first processor to the second processor. 

25 . The system of claim 2 1 , further comprising: 

a storage medium coupled to said first input, that stores each of said at least one 
reference signals to be monitored; and 

a controller that compares an abstract for each reference signal being input for the first 
time to be compared to all previously stored abstracts in the reference database, such that in 
the event that the comparing device determines that it cannot distinguish between the abstract 
of a reference signal being input for the first time from a previously stored abstract in the 
reference database, the controller adjusts the criteria being used by the processor and re- 
generates the reference database, by re-processing each reference signal stored on the storage 
medium to create new abstracts and storing said new abstracts in the reference database. 
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METHOD AND DEVICE FOR MONITORING AND ANALYZING SIGNALS 



ABSTRACT OF THE DISCLOSURE 

A method and system for monitoring and analyzing at least one signal are disclosed. An 
abstract of at least one reference signal is generated and stored in a reference database. An 
abstract of a query signal to be analyzed is then generated so that the abstract of the query signal 
can be compared to the abstracts stored in the reference database for a match. The method and 
system may optionally be used to record information about the query signals, the number of 
matches recorded, and other useful information about the query signals. Moreover, the method 
by which abstracts are generated can be programmable based upon selectable criteria. The 
system can also be programmed with error control software so as to avoid the re-occurrence of a 
query signal that matches more than one signal stored in the reference database. 
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